<!DOCTYPE html>
<html lang="zh-CN" class="loading">

<head>
    <meta charset="UTF-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
    <meta name="viewport" content="width=device-width, minimum-scale=1.0, maximum-scale=1.0, user-scalable=no">
    <title>scrapy+GitHubActions部署云爬虫 - Skyone-Blog</title>
    <meta name="apple-mobile-web-app-capable" content="yes" />
    <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent">
    <meta name="google" content="notranslate" />
    <meta name="keywords" content="爬虫,Scrapy,Github,"> 
    <meta name="description" content="Skyone技术博客,为了学习GitHub Actions的基本使用，写一个爬虫练练手，这个爬虫用于下载哔哩哔哩相簿的图片，该文章分为三步：写爬虫、写workflows文件、上传代码到GitHub"> 
    <meta name="author" content="王志伟"> 
    <link rel="alternative" href="atom.xml" title="Skyone-Blog" type="application/atom+xml"> 
    <link rel="icon" href="/img/favicon.png"> 
    
<link rel="stylesheet" href="//cdn.jsdelivr.net/npm/gitalk@1/dist/gitalk.css">

    
<link rel="stylesheet" href="/css/diaspora.css">

    <!-- 看板娘 -->
    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/font-awesome/css/font-awesome.min.css">
    <script src="/live2d-widget/autoload.js"></script>
    <!-- MD5 -->
    <script src="/js/md5.min.js"></script>
    <!-- MathJax -->
    
        <script type="text/x-mathjax-config">
        (function () {
            let target = document.getElementsByClassName("content")[0];
            MathJax.Hub.Config({
                tex2jax: {
                    inlineMath:  [ ["$", "$"] ],
                    displayMath: [ ["$$","$$"] ],
                    skipTags: ['script', 'noscript', 'style', 'textarea', 'pre','code','a'],
                    ignoreClass:"class1"
                }
            });
            MathJax.Hub.Queue(["Typeset",MathJax.Hub,target]);
        })();
    </script>
<script src="https://polyfill.io/v3/polyfill.min.js?features=es6"></script>
<script id="MathJax-script" async src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>

    
<meta name="generator" content="Hexo 5.3.0"></head>

<body class="loading">
    <span id="config-title" style="display:none">Skyone-Blog</span>
    <div id="loader"></div>
    <div id="single">
    <div id="top" style="display: block;">
    <div class="bar" style="width: 0;"></div>
    <a class="iconfont icon-home image-icon" href="javascript:;" data-url="https://skyone-wzw.gitee.io"></a>
    <div title="播放/暂停" class="iconfont icon-play"></div>
    <h3 class="subtitle">scrapy+GitHubActions部署云爬虫</h3>
    <div class="social">
        <div>
            <div class="share">
                <a title="获取二维码" class="iconfont icon-scan" href="javascript:;"></a>
            </div>
            <div id="qr"></div>
        </div>
    </div>
    <div class="scrollbar"></div>
</div>

    <div class="section">
        <div class="article">
    <div class='main'>
        <h1 class="title">scrapy+GitHubActions部署云爬虫</h1>
        <div class="stuff">
            <span>三月 21, 2021</span>
            
  <ul class="post-tags-list" itemprop="keywords"><li class="post-tags-list-item"><a class="post-tags-list-link" href="/tags/Github/" rel="tag">Github</a></li><li class="post-tags-list-item"><a class="post-tags-list-link" href="/tags/%E7%88%AC%E8%99%AB/" rel="tag">爬虫</a></li></ul>


        </div>
        <div class="content markdown">
            <p>为了学习GitHub Actions的基本使用，写一个爬虫练练手</p>
<p>这个爬虫用于下载哔哩哔哩相簿的图片</p>
<p>该文章分为三步：</p>
<ul>
<li> 写爬虫</li>
<li>写workflows文件</li>
<li>上传代码到GitHub</li>
</ul>
<h2 id="写一个scrapy爬虫"><a href="#写一个scrapy爬虫" class="headerlink" title="写一个scrapy爬虫"></a>写一个scrapy爬虫</h2><h3 id="创建项目"><a href="#创建项目" class="headerlink" title="创建项目"></a>创建项目</h3><p>新建一个项目文件夹，打开CMD并切换到该文件夹，输入以下命令初始化git：</p>
<figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">git init</span><br></pre></td></tr></table></figure>
<p>使用scrapy新建项目并新建爬虫：</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">scrapy startproject bili bilibili.com</span><br><span class="line">cd bili</span><br><span class="line">scrapy genspider picture</span><br></pre></td></tr></table></figure>
<p>创建项目完毕，下面使用IDE打开项目文件夹吧~</p>
<p>我使用的是Pycharm</p>
<h3 id="储存图片信息的class"><a href="#储存图片信息的class" class="headerlink" title="储存图片信息的class"></a>储存图片信息的class</h3><p>打开<code>bili/items.py</code>，不出意外的话，这里已经有一个<code>BiliItem(scrapy.Item)</code>类了</p>
<p>我们将其改为：</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> scrapy</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">BiliItem</span>(<span class="params">scrapy.Item</span>):</span></span><br><span class="line">    url = scrapy.Field()		<span class="comment"># 图片连接</span></span><br><span class="line">    title = scrapy.Field()		<span class="comment"># 图片标题</span></span><br><span class="line">    author = scrapy.Field()		<span class="comment"># 作者</span></span><br><span class="line">    <span class="built_in">id</span> = scrapy.Field()			<span class="comment"># 图片id</span></span><br><span class="line">    uid = scrapy.Field()		<span class="comment"># 作者uid</span></span><br><span class="line">    extension = scrapy.Field()	<span class="comment"># 图片拓展名</span></span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h3 id="分析哔哩哔哩API"><a href="#分析哔哩哔哩API" class="headerlink" title="分析哔哩哔哩API"></a>分析哔哩哔哩API</h3><p>打开<a target="_blank" rel="noopener" href="https://h.bilibili.com/eden/draw_area">哔哩哔哩相簿</a>，开启检查元素的网络那一栏，发现每加载一次都会有一个get请求：</p>
<p><img src="https://tc.skyone.host/blog/post/scrapy-GitHubActions%E9%83%A8%E7%BD%B2%E4%BA%91%E7%88%AC%E8%99%AB/%E8%AF%B7%E6%B1%82%E5%9B%BE%E7%89%87.png" alt="请求图片"></p>
<p>很好，正是我们想要的，多次尝试，很容易分析出来：</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">GET https://api.vc.bilibili.com/link_draw/v2/Doc/list?category=all&amp;type=hot&amp;page_num=0&amp;page_size=20</span><br><span class="line">Accept: application/json</span><br></pre></td></tr></table></figure>
<p>返回：</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br></pre></td><td class="code"><pre><span class="line">GET https://api.vc.bilibili.com/link_draw/v2/Doc/list?category=all&amp;type=hot&amp;page_num=0&amp;page_size=20</span><br><span class="line"></span><br><span class="line"><span class="meta">HTTP/1.1</span> <span class="number">200</span> OK</span><br><span class="line"><span class="attribute">Date</span><span class="punctuation">: </span>Sun, 21 Mar 2021 06:30:48 GMT</span><br><span class="line"><span class="attribute">Content-Type</span><span class="punctuation">: </span>application/json</span><br><span class="line"><span class="attribute">Transfer-Encoding</span><span class="punctuation">: </span>chunked</span><br><span class="line"><span class="attribute">Connection</span><span class="punctuation">: </span>keep-alive</span><br><span class="line"><span class="attribute">L</span><span class="punctuation">: </span>v</span><br><span class="line"><span class="attribute">X-Trace-Id</span><span class="punctuation">: </span>29bd4f5ef5186679</span><br><span class="line"><span class="attribute">Http-Status-Code</span><span class="punctuation">: </span>200</span><br><span class="line"><span class="attribute">Bili-Status-Code</span><span class="punctuation">: </span>0</span><br><span class="line"><span class="attribute">Server</span><span class="punctuation">: </span>swoole-http-server</span><br><span class="line"><span class="attribute">Set-Cookie</span><span class="punctuation">: </span>l=v</span><br><span class="line"><span class="attribute">Expires</span><span class="punctuation">: </span>Sun, 21 Mar 2021 06:30:47 GMT</span><br><span class="line"><span class="attribute">Cache-Control</span><span class="punctuation">: </span>no-cache</span><br><span class="line"><span class="attribute">X-Cache-Webcdn</span><span class="punctuation">: </span>BYPASS from hw-sh3-webcdn-08</span><br><span class="line"></span><br><span class="line">&#123;</span><br><span class="line">  &quot;code&quot;: 0,</span><br><span class="line">  &quot;msg&quot;: &quot;success&quot;,</span><br><span class="line">  &quot;message&quot;: &quot;success&quot;,</span><br><span class="line">  &quot;data&quot;: &#123;</span><br><span class="line">    &quot;total_count&quot;: 500,</span><br><span class="line">    &quot;items&quot;: [</span><br><span class="line">      &#123;</span><br><span class="line">        &quot;user&quot;: &#123;</span><br><span class="line">          &quot;uid&quot;: 21690338,</span><br><span class="line">          &quot;head_url&quot;: &quot;https://i1.hdslb.com/bfs/face/b6a132c21e444401228099c8cc07edab810bc9db.jpg&quot;,</span><br><span class="line">          &quot;name&quot;: &quot;ZWPAN盼&quot;</span><br><span class="line">        &#125;,</span><br><span class="line">        &quot;item&quot;: &#123;</span><br><span class="line">          &quot;doc_id&quot;: 1085297,</span><br><span class="line">          &quot;poster_uid&quot;: 21690338,</span><br><span class="line">          &quot;pictures&quot;: [</span><br><span class="line">            &#123;</span><br><span class="line">              &quot;img_src&quot;: &quot;https://i0.hdslb.com/bfs/album/2785de69bb019f85c88ee0d9681468c779e3950f.jpg&quot;,</span><br><span class="line">              &quot;img_width&quot;: 3029,</span><br><span class="line">              &quot;img_height&quot;: 9506,</span><br><span class="line">              &quot;img_size&quot;: 6425</span><br><span class="line">            &#125;</span><br><span class="line">          ],</span><br><span class="line">          &quot;title&quot;: &quot;108个小电视表情&quot;,</span><br><span class="line">          &quot;category&quot;: &quot;illustration&quot;,</span><br><span class="line">          &quot;upload_time&quot;: 1512023535,</span><br><span class="line">          &quot;already_liked&quot;: 0,</span><br><span class="line">          &quot;already_voted&quot;: 0</span><br><span class="line">        &#125;</span><br><span class="line">      &#125;,</span><br><span class="line">//////////////////////////////////</span><br><span class="line">/     &#123;							 /</span><br><span class="line">/     &quot;_&quot;: &quot;太多了，就不列举了&quot;     /</span><br><span class="line">/     &#125;							 /</span><br><span class="line">//////////////////////////////////</span><br><span class="line">      ]</span><br><span class="line">  &#125;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">Response code: 200 (OK); Time: 490ms; Content length: 12879 bytes</span><br></pre></td></tr></table></figure>
<ul>
<li><p><code>URL</code>：<code>https://api.vc.bilibili.com/link_draw/v2/Doc/list</code></p>
</li>
<li><p><code>method</code>：<code>GET</code></p>
</li>
<li><p><code>params</code>：</p>
<ul>
<li><p><code>category</code></p>
<p>图片的类型，选项如下</p>
<ul>
<li><code>all</code>：所有类型</li>
<li><code>illustration</code>：插画</li>
<li><code>comic</code>：漫画</li>
<li><code>draw</code>：其他</li>
</ul>
</li>
<li><p><code>type</code></p>
<p>排名规则。选项如下：</p>
<ul>
<li><code>hot</code>：按热度排序</li>
<li><code>new</code>：按时间排序</li>
</ul>
</li>
<li><p><code>page_num</code></p>
<p>页数，从0开始</p>
</li>
<li><p><code>page_size</code></p>
<p>每页图片数</p>
</li>
</ul>
</li>
</ul>
<h3 id="写爬虫"><a href="#写爬虫" class="headerlink" title="写爬虫"></a>写爬虫</h3><p>先在项目的根目录创建一个配置文件<code>setting.py</code></p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># picture settings</span></span><br><span class="line"></span><br><span class="line">PICTURE_MAX_PAGE = <span class="number">20</span></span><br><span class="line">PICTURE_SLEEP_TIME = <span class="number">0.5</span></span><br><span class="line">PICTURE_CATEGORY = <span class="string">&quot;all&quot;</span></span><br><span class="line">PICTURE_TYPE = <span class="string">&quot;hot&quot;</span></span><br></pre></td></tr></table></figure>
<ul>
<li><p><code>PICTURE_MAX_PAGE</code></p>
<p>爬的页数，每页20个</p>
</li>
<li><p><code>PICTURE_SELLP_TIME</code></p>
<p>每爬一张图间隔的时间</p>
</li>
<li><p><code>PICTURE_CATEGORY</code></p>
<p>要爬的图片的类型，选项如下</p>
<ul>
<li><code>all</code>：所有类型</li>
<li><code>illustration</code>：插画</li>
<li><code>comic</code>：漫画</li>
<li><code>draw</code>：其他</li>
</ul>
</li>
<li><p><code>PICTURE_TYPE</code></p>
<p>排名规则。选项如下：</p>
<ul>
<li><code>hot</code>：按热度排序</li>
<li><code>new</code>：按时间排序</li>
</ul>
</li>
</ul>
<hr>
<p>打开<code>bili/spider/picture.py</code></p>
<p>导入一些库</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> json</span><br><span class="line"><span class="keyword">import</span> scrapy</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="keyword">import</span> setting</span><br><span class="line"><span class="keyword">from</span> bili.items <span class="keyword">import</span> BiliItem</span><br></pre></td></tr></table></figure>
<p>改写<code>class PictureSpider(scrapy.Spider):</code></p>
<p>我们要先从<code>https://h.bilibili.com/</code>开始爬，从而免去设置<code>Referer</code></p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">start_urls = [<span class="string">&#x27;https://h.bilibili.com/&#x27;</span>]</span><br></pre></td></tr></table></figure>
<p>再接着循环爬取刚才分析的链接即可，记住要构造请求并设置回调函数哦！</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">parse</span>(<span class="params">self, response, **kwargs</span>):</span></span><br><span class="line">    <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(setting.PICTURE_MAX_PAGE):</span><br><span class="line">        <span class="keyword">yield</span> scrapy.Request(</span><br><span class="line">            <span class="string">&#x27;https://api.vc.bilibili.com/link_draw/v2/Doc/list?page_size=20&#x27;</span></span><br><span class="line">            <span class="string">&#x27;&amp;type=&#x27;</span> + setting.PICTURE_TYPE +</span><br><span class="line">            <span class="string">&#x27;&amp;category=&#x27;</span> + setting.PICTURE_CATEGORY +</span><br><span class="line">            <span class="string">&#x27;&amp;page_num=&#x27;</span> + <span class="built_in">str</span>(i),</span><br><span class="line">            callback=self.picture_info,</span><br><span class="line">            dont_filter=<span class="literal">True</span></span><br><span class="line">        )</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">picture_info</span>(<span class="params">self, response, **kwargs</span>):</span></span><br><span class="line">    data = json.loads(response.text)</span><br><span class="line">    <span class="keyword">for</span> item <span class="keyword">in</span> data[<span class="string">&quot;data&quot;</span>][<span class="string">&quot;items&quot;</span>]:</span><br><span class="line">        img = BiliItem()</span><br><span class="line">        img[<span class="string">&quot;url&quot;</span>] = item[<span class="string">&quot;item&quot;</span>][<span class="string">&quot;pictures&quot;</span>][-<span class="number">1</span>][<span class="string">&quot;img_src&quot;</span>]</span><br><span class="line">        img[<span class="string">&quot;title&quot;</span>] = item[<span class="string">&quot;item&quot;</span>][<span class="string">&quot;title&quot;</span>]</span><br><span class="line">        img[<span class="string">&quot;id&quot;</span>] = item[<span class="string">&quot;item&quot;</span>][<span class="string">&quot;doc_id&quot;</span>]</span><br><span class="line">        img[<span class="string">&quot;author&quot;</span>] = item[<span class="string">&quot;user&quot;</span>][<span class="string">&quot;name&quot;</span>]</span><br><span class="line">        img[<span class="string">&quot;uid&quot;</span>] = item[<span class="string">&quot;user&quot;</span>][<span class="string">&quot;uid&quot;</span>]</span><br><span class="line">        img[<span class="string">&quot;extension&quot;</span>] = img[<span class="string">&quot;url&quot;</span>].split(<span class="string">&#x27;.&#x27;</span>)[-<span class="number">1</span>]</span><br><span class="line">        <span class="keyword">yield</span> img</span><br><span class="line">        time.sleep(setting.PICTURE_SLEEP_TIME)</span><br></pre></td></tr></table></figure>
<p><code>bili/spiders/picture.py</code>完整代码</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> json</span><br><span class="line"><span class="keyword">import</span> scrapy</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="keyword">import</span> setting</span><br><span class="line"><span class="keyword">from</span> bili.items <span class="keyword">import</span> BiliItem</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">PictureSpider</span>(<span class="params">scrapy.Spider</span>):</span></span><br><span class="line">    name = <span class="string">&#x27;picture&#x27;</span></span><br><span class="line">    allowed_domains = [<span class="string">&#x27;bilibili.com&#x27;</span>]</span><br><span class="line">    start_urls = [<span class="string">&#x27;https://h.bilibili.com/&#x27;</span>]</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">parse</span>(<span class="params">self, response, **kwargs</span>):</span></span><br><span class="line">        <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(setting.PICTURE_MAX_PAGE):</span><br><span class="line">            <span class="keyword">yield</span> scrapy.Request(</span><br><span class="line">                <span class="string">&#x27;https://api.vc.bilibili.com/link_draw/v2/Doc/list?page_size=20&#x27;</span></span><br><span class="line">                <span class="string">&#x27;&amp;type=&#x27;</span> + setting.PICTURE_TYPE +</span><br><span class="line">                <span class="string">&#x27;&amp;category=&#x27;</span> + setting.PICTURE_CATEGORY +</span><br><span class="line">                <span class="string">&#x27;&amp;page_num=&#x27;</span> + <span class="built_in">str</span>(i),</span><br><span class="line">                callback=self.picture_info,</span><br><span class="line">                dont_filter=<span class="literal">True</span></span><br><span class="line">            )</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">picture_info</span>(<span class="params">self, response, **kwargs</span>):</span></span><br><span class="line">        data = json.loads(response.text)</span><br><span class="line">        <span class="keyword">for</span> item <span class="keyword">in</span> data[<span class="string">&quot;data&quot;</span>][<span class="string">&quot;items&quot;</span>]:</span><br><span class="line">            img = BiliItem()</span><br><span class="line">            img[<span class="string">&quot;url&quot;</span>] = item[<span class="string">&quot;item&quot;</span>][<span class="string">&quot;pictures&quot;</span>][-<span class="number">1</span>][<span class="string">&quot;img_src&quot;</span>]</span><br><span class="line">            img[<span class="string">&quot;title&quot;</span>] = item[<span class="string">&quot;item&quot;</span>][<span class="string">&quot;title&quot;</span>]</span><br><span class="line">            img[<span class="string">&quot;id&quot;</span>] = item[<span class="string">&quot;item&quot;</span>][<span class="string">&quot;doc_id&quot;</span>]</span><br><span class="line">            img[<span class="string">&quot;author&quot;</span>] = item[<span class="string">&quot;user&quot;</span>][<span class="string">&quot;name&quot;</span>]</span><br><span class="line">            img[<span class="string">&quot;uid&quot;</span>] = item[<span class="string">&quot;user&quot;</span>][<span class="string">&quot;uid&quot;</span>]</span><br><span class="line">            img[<span class="string">&quot;extension&quot;</span>] = img[<span class="string">&quot;url&quot;</span>].split(<span class="string">&#x27;.&#x27;</span>)[-<span class="number">1</span>]</span><br><span class="line">            <span class="keyword">yield</span> img</span><br><span class="line">            time.sleep(setting.PICTURE_SLEEP_TIME)</span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h3 id="下载图片"><a href="#下载图片" class="headerlink" title="下载图片"></a>下载图片</h3><p>图片链接已经使用上面的<code>picture_info()</code>方法获取了，下面使用<code>BiliPipeline</code>管道下载图片</p>
<p>打开<code>bili/pipelines.py</code></p>
<p>由于scrapy自带的图片下载器个人觉得使用起来太麻烦，这里直接用requests库就好</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"><span class="keyword">from</span> os <span class="keyword">import</span> path</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">BiliPipeline</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">process_item</span>(<span class="params">self, item, spider</span>):</span></span><br><span class="line">        name = <span class="built_in">str</span>(item[<span class="string">&quot;id&quot;</span>]) + <span class="string">&#x27;.&#x27;</span> + item[<span class="string">&quot;extension&quot;</span>]</span><br><span class="line">        res = requests.get(item[<span class="string">&quot;url&quot;</span>])</span><br><span class="line">        <span class="keyword">with</span> <span class="built_in">open</span>(path.join(path.join(path.abspath(<span class="string">&#x27;.&#x27;</span>), <span class="string">&#x27;image&#x27;</span>), name), <span class="string">&#x27;wb&#x27;</span>) <span class="keyword">as</span> file:</span><br><span class="line">            file.write(res.content)</span><br><span class="line">            print(name)</span><br><span class="line">        <span class="keyword">return</span> item</span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h3 id="启用管道"><a href="#启用管道" class="headerlink" title="启用管道"></a>启用管道</h3><p>打开<code>bili/settings.py</code></p>
<p>找到以下内容并取消注释</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">ITEM_PIPELINES = &#123;</span><br><span class="line">   <span class="string">&#x27;bili.pipelines.BiliPipeline&#x27;</span>: <span class="number">300</span>,</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>
<blockquote>
<p> 在运行阶段为了减少不必要的log提示，可以在在里面添加</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">LOG_LEVEL = <span class="string">&quot;WARNING&quot;</span></span><br></pre></td></tr></table></figure>
</blockquote>
<h3 id="本地启动爬虫测试"><a href="#本地启动爬虫测试" class="headerlink" title="本地启动爬虫测试"></a>本地启动爬虫测试</h3><h4 id="启动爬虫"><a href="#启动爬虫" class="headerlink" title="启动爬虫"></a>启动爬虫</h4><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash"> 切换到项目根目录</span></span><br><span class="line">scrapy crawl picture</span><br></pre></td></tr></table></figure>
<h4 id="pycharm调试爬虫"><a href="#pycharm调试爬虫" class="headerlink" title="pycharm调试爬虫"></a>pycharm调试爬虫</h4><p>点击右上角“编辑配置”</p>
<img src="https://tc.skyone.host/blog/post/scrapy-GitHubActions%E9%83%A8%E7%BD%B2%E4%BA%91%E7%88%AC%E8%99%AB/%E7%BC%96%E8%BE%91%E9%85%8D%E7%BD%AE.png" alt="编辑配置" style="zoom: 67%;" />

<p>添加配置，选择python</p>
<img src="https://tc.skyone.host/blog/post/scrapy-GitHubActions%E9%83%A8%E7%BD%B2%E4%BA%91%E7%88%AC%E8%99%AB/%E6%B7%BB%E5%8A%A0%E9%85%8D%E7%BD%AE.png" alt="添加配置" style="zoom:67%;" />

<p>配置如下</p>
<img src="https://tc.skyone.host/blog/post/scrapy-GitHubActions%E9%83%A8%E7%BD%B2%E4%BA%91%E7%88%AC%E8%99%AB/%E9%85%8D%E7%BD%AE.png" alt="配置" style="zoom:80%;" />

<ul>
<li>脚本路径为<code>python安装目录\Lib\site-packages\scrapy\cmdline.py</code></li>
<li>参数为<code>crawl 爬虫名</code></li>
<li>工作目录为项目根目录</li>
</ul>
<p>然后就可以愉快的打断点调试了<del>JetBrains NB</del></p>
<h2 id="写workflows文件"><a href="#写workflows文件" class="headerlink" title="写workflows文件"></a>写workflows文件</h2><p>在项目根目录创建：<code>.github/workflows/blank.yml</code></p>
<p>直接复制粘贴即可，有时间我会另写一篇入门GitHub Actions的文章，<del>可能会鸽，咕咕咕。。。其实网上有很多这样的文章，例如：阮一峰的网络日志</del></p>
<figure class="highlight yml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">name:</span> <span class="string">Spider</span></span><br><span class="line"></span><br><span class="line"><span class="attr">on:</span></span><br><span class="line">  <span class="attr">workflow_dispatch:</span></span><br><span class="line"></span><br><span class="line"><span class="attr">jobs:</span></span><br><span class="line">  <span class="attr">spider:</span></span><br><span class="line">    <span class="attr">runs-on:</span> <span class="string">ubuntu-latest</span></span><br><span class="line">    <span class="attr">steps:</span></span><br><span class="line">    <span class="bullet">-</span> <span class="attr">name:</span> <span class="string">checkout</span></span><br><span class="line">      <span class="attr">uses:</span> <span class="string">actions/checkout@master</span></span><br><span class="line"></span><br><span class="line">    <span class="bullet">-</span> <span class="attr">name:</span> <span class="string">&#x27;Set up Python&#x27;</span></span><br><span class="line">      <span class="attr">uses:</span> <span class="string">actions/setup-python@v1</span></span><br><span class="line">      <span class="attr">with:</span></span><br><span class="line">        <span class="attr">python-version:</span> <span class="number">3.7</span></span><br><span class="line"></span><br><span class="line">    <span class="bullet">-</span> <span class="attr">name:</span> <span class="string">Run</span> <span class="string">a</span> <span class="string">single-line</span> <span class="string">script</span></span><br><span class="line">      <span class="attr">run:</span> <span class="string">|</span></span><br><span class="line">        <span class="string">pip</span> <span class="string">install</span> <span class="string">requests</span></span><br><span class="line">        <span class="string">pip</span> <span class="string">install</span> <span class="string">lxml</span></span><br><span class="line">        <span class="string">pip</span> <span class="string">install</span> <span class="string">scrapy</span></span><br><span class="line">        <span class="string">scrapy</span> <span class="string">crawl</span> <span class="string">picture</span></span><br><span class="line"></span><br><span class="line">    <span class="bullet">-</span> <span class="attr">name:</span> <span class="string">Upload</span> <span class="string">artifact</span></span><br><span class="line">      <span class="attr">uses:</span> <span class="string">actions/upload-artifact@master</span></span><br><span class="line">      <span class="attr">with:</span></span><br><span class="line">        <span class="attr">name:</span> <span class="string">img</span></span><br><span class="line">        <span class="attr">path:</span> <span class="string">image</span></span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h2 id="部署到GitHub-Actions"><a href="#部署到GitHub-Actions" class="headerlink" title="部署到GitHub Actions"></a>部署到GitHub Actions</h2><p>在Github上新建仓库</p>
<p>将代码推送到该仓库里</p>
<p>Git三部曲：</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">git add .				# 添加</span><br><span class="line">git commit -m &quot;init&quot;	# 提交</span><br><span class="line">git push				# 推送</span><br></pre></td></tr></table></figure>
<p>打开GitHub仓库网页</p>
<p>依次点击：</p>
<p><img src="https://tc.skyone.host/blog/post/scrapy-GitHubActions%E9%83%A8%E7%BD%B2%E4%BA%91%E7%88%AC%E8%99%AB/action.png" alt="action"></p>
<p>等待一小会，不出意外的话，爬虫的结果就显示出来了，点击下载即可</p>
<p><img src="https://tc.skyone.host/blog/post/scrapy-GitHubActions%E9%83%A8%E7%BD%B2%E4%BA%91%E7%88%AC%E8%99%AB/%E4%B8%8B%E8%BD%BD.png" alt="下载"></p>

            <!--[if lt IE 9]><script>document.createElement('audio');</script><![endif]-->
            <audio id="audio" loop="1" preload="auto" controls="controls" data-autoplay="false">
                <source type="audio/mpeg" src="">
            </audio>
            
                <ul id="audio-list" style="display:none">
                    
                        
                            <li title='0' data-url='https://tc.skyone.host/blog/music/%E6%B4%9B%E5%A4%A9%E4%BE%9D%20-%20%E5%A4%9C%E8%88%AA%E6%98%9F.flac'></li>
                        
                    
                        
                            <li title='1' data-url='https://tc.skyone.host/blog/music/%E6%B4%9B%E5%A4%A9%E4%BE%9D%20-%20%E8%8C%89%E8%8E%89%E8%8A%B1.mp3'></li>
                        
                    
                        
                            <li title='2' data-url='https://tc.skyone.host/blog/music/%E8%B5%A4%E7%BE%BD%20-%20%E4%B8%87%E5%8F%A4%E7%94%9F%E9%A6%99.flac'></li>
                        
                    
                        
                            <li title='3' data-url='https://tc.skyone.host/blog/music/%E6%B4%9B%E5%A4%A9%E4%BE%9D%20-%20%E4%B8%BA%E8%B0%81%E8%80%8C%E4%B8%BA.mp3'></li>
                        
                    
                        
                            <li title='4' data-url='https://tc.skyone.host/blog/music/%E4%B9%90%E6%AD%A3%E7%BB%AB%20-%20Faded.mp3'></li>
                        
                    
                        
                            <li title='5' data-url='https://tc.skyone.host/blog/music/bilibili2018%20-%20%E8%87%B3%E5%B0%8A%E9%A9%AC%E7%94%B2.flac'></li>
                        
                    
                </ul>
            
        </div>
        
    <div id='gitalk-container' class="comment link"
		data-enable='true'
        data-ae='true'
        data-ci='d5e197ef955b7f3268e5'
        data-cs='7d4feed7a179ad28943b0865b7970814073145ad'
        data-r='blog-gitalk'
        data-o='skyone-wzw'
        data-a='skyone-wzw'
        data-d='false'
    >查看评论</div>


    </div>
    
        <div class='side'>
			<ol class="toc"><li class="toc-item toc-level-2"><a class="toc-link" href="#%E5%86%99%E4%B8%80%E4%B8%AAscrapy%E7%88%AC%E8%99%AB"><span class="toc-number">1.</span> <span class="toc-text">写一个scrapy爬虫</span></a><ol class="toc-child"><li class="toc-item toc-level-3"><a class="toc-link" href="#%E5%88%9B%E5%BB%BA%E9%A1%B9%E7%9B%AE"><span class="toc-number">1.1.</span> <span class="toc-text">创建项目</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#%E5%82%A8%E5%AD%98%E5%9B%BE%E7%89%87%E4%BF%A1%E6%81%AF%E7%9A%84class"><span class="toc-number">1.2.</span> <span class="toc-text">储存图片信息的class</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#%E5%88%86%E6%9E%90%E5%93%94%E5%93%A9%E5%93%94%E5%93%A9API"><span class="toc-number">1.3.</span> <span class="toc-text">分析哔哩哔哩API</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#%E5%86%99%E7%88%AC%E8%99%AB"><span class="toc-number">1.4.</span> <span class="toc-text">写爬虫</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#%E4%B8%8B%E8%BD%BD%E5%9B%BE%E7%89%87"><span class="toc-number">1.5.</span> <span class="toc-text">下载图片</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#%E5%90%AF%E7%94%A8%E7%AE%A1%E9%81%93"><span class="toc-number">1.6.</span> <span class="toc-text">启用管道</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#%E6%9C%AC%E5%9C%B0%E5%90%AF%E5%8A%A8%E7%88%AC%E8%99%AB%E6%B5%8B%E8%AF%95"><span class="toc-number">1.7.</span> <span class="toc-text">本地启动爬虫测试</span></a><ol class="toc-child"><li class="toc-item toc-level-4"><a class="toc-link" href="#%E5%90%AF%E5%8A%A8%E7%88%AC%E8%99%AB"><span class="toc-number">1.7.1.</span> <span class="toc-text">启动爬虫</span></a></li><li class="toc-item toc-level-4"><a class="toc-link" href="#pycharm%E8%B0%83%E8%AF%95%E7%88%AC%E8%99%AB"><span class="toc-number">1.7.2.</span> <span class="toc-text">pycharm调试爬虫</span></a></li></ol></li></ol></li><li class="toc-item toc-level-2"><a class="toc-link" href="#%E5%86%99workflows%E6%96%87%E4%BB%B6"><span class="toc-number">2.</span> <span class="toc-text">写workflows文件</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#%E9%83%A8%E7%BD%B2%E5%88%B0GitHub-Actions"><span class="toc-number">3.</span> <span class="toc-text">部署到GitHub Actions</span></a></li></ol>	
        </div>
    
</div>


    </div>
</div>
</body>

<script src="//cdn.jsdelivr.net/npm/gitalk@1/dist/gitalk.min.js"></script>


<script src="//lib.baomitu.com/jquery/1.8.3/jquery.min.js"></script>
<script src="/js/plugin.js"></script>
<script src="/js/typed.js"></script>
<script src="/js/skyone.min.js"></script>


<link rel="stylesheet" href="/photoswipe/photoswipe.css">
<link rel="stylesheet" href="/photoswipe/default-skin/default-skin.css">


<script src="/photoswipe/photoswipe.min.js"></script>
<script src="/photoswipe/photoswipe-ui-default.min.js"></script>


<!-- Root element of PhotoSwipe. Must have class pswp. -->
<div class="pswp" tabindex="-1" role="dialog" aria-hidden="true">
    <!-- Background of PhotoSwipe. 
         It's a separate element as animating opacity is faster than rgba(). -->
    <div class="pswp__bg"></div>
    <!-- Slides wrapper with overflow:hidden. -->
    <div class="pswp__scroll-wrap">
        <!-- Container that holds slides. 
            PhotoSwipe keeps only 3 of them in the DOM to save memory.
            Don't modify these 3 pswp__item elements, data is added later on. -->
        <div class="pswp__container">
            <div class="pswp__item"></div>
            <div class="pswp__item"></div>
            <div class="pswp__item"></div>
        </div>
        <!-- Default (PhotoSwipeUI_Default) interface on top of sliding area. Can be changed. -->
        <div class="pswp__ui pswp__ui--hidden">
            <div class="pswp__top-bar">
                <!--  Controls are self-explanatory. Order can be changed. -->
                <div class="pswp__counter"></div>
                <button class="pswp__button pswp__button--close" title="Close (Esc)"></button>
                <button class="pswp__button pswp__button--share" title="Share"></button>
                <button class="pswp__button pswp__button--fs" title="Toggle fullscreen"></button>
                <button class="pswp__button pswp__button--zoom" title="Zoom in/out"></button>
                <!-- Preloader demo http://codepen.io/dimsemenov/pen/yyBWoR -->
                <!-- element will get class pswp__preloader--active when preloader is running -->
                <div class="pswp__preloader">
                    <div class="pswp__preloader__icn">
                      <div class="pswp__preloader__cut">
                        <div class="pswp__preloader__donut"></div>
                      </div>
                    </div>
                </div>
            </div>
            <div class="pswp__share-modal pswp__share-modal--hidden pswp__single-tap">
                <div class="pswp__share-tooltip"></div> 
            </div>
            <button class="pswp__button pswp__button--arrow--left" title="Previous (arrow left)">
            </button>
            <button class="pswp__button pswp__button--arrow--right" title="Next (arrow right)">
            </button>
            <div class="pswp__caption">
                <div class="pswp__caption__center"></div>
            </div>
        </div>
    </div>
</div>




</html>
